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FOREWORD 


Requirement : 


To  investigate  and  determine  the  effectiveness  of  current  prac¬ 
tices  and  procedures  used  by  service  schools  to  validate  TEC  lessons . 


Procedure : 

During  the  period  August  to  November  1978,  TEC  validations  being 
conducted  by  five  different  service  schools  were  observed. 


Findings : 

The  validation  guidance  contained  in  the  TEC  Manager's  Guidebook 
is  used  as  the  basis  for  planning  TEC  validations.  Several  deviations 
from  the  prescribed  procedures  were  observed;  however,  these  adjust¬ 
ments  were  the  results  of  conscious  decisions  made  with  the  intent  of 
facilitating  mission  accomplishment  with  available  resources.  The 
lack  of  subject  matter  expert  (SME)  influence  during  lesson  develop¬ 
ment  and  the  failure  to  use  hands-on  performance  tests  in  validations 
has  an  adverse  impact  on  the  quality  of  completed  TEC  lessons. 


Utilization: 

This  report  describes  several  shortcomings  associated  with  the 
currant  practices  and  procedures  used  in  TEC  lesson  validations  and 
presents  suggested  modifications. 
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TRAINING  EXTENSION  COURSE  VALIDATION 


INTRODUCTION 


GENERAL 

The  Training  Extension  Course  (TEC)  program  has  been  designed  to 
put  into  the  hands  of  trainers,  both  in  units  and  in  institutions, 
high  quality  performance-oriented  multimedia  training  packages.  It 
is  designed  to  provide  soldiers  with  immediate  access  to  self-paced 
instruction  especially  designed  to  assist  them  in  acquiring  and  main¬ 
taining  skills  critical  to  the  performance  of  their  job  in  combat. 

TEC  lessons  are  designed  for  use  on  an  individual  basis;  however, 
they  may  be  used  by  small  groups,  under  the  supervision  of  an  NCO. 

The  advent  of  the  TEC  program  inaugurated  a  multi-faceted  TEC 
research  effort  conducted  under  the  sponsorship  of  the  U.  S.  Army 
Training  Support  Center  (USATSC),  by  the  U.  S.  Army  Research  Insti¬ 
tute  for  the  Behavioral  and  Social  Sciences  (ARI)  and  the  Mellonics 
Systems  Development  Division  of  Litton  Systems,  Inc.  The  research 
comprises  investigations  concerning  current  and  projected  usage  of 
TEC  in  the  Active  and  Reserve  Components,  training  effectiveness  and 
retention  of  TEC  instructions,  current  and  projected  costs  of  the  TEC 
program,  cost-effectiveness  of  TEC  as  it  relates  to  individual  and 
unit  performance,  and  the  development  of  a  TEC  Manager’s  Guidebook. 

The  requirements  of  this  effort  are  to  determine  the  current 
practices  and  procedures  used  by  the  service  schools  to  validate  TEC 
lessons.  As  a  basis  for  discussing  TEC  lesson  validations,  the 
validation  chapter  of  the  U.  S.  Army  Training  Extension  Course 
Manager ' s  Guidebook  is  reviewed. 


VALIDATION  GUIDANCE 

The  TEC  Manager's  Guidebook  provides  general  procedural  guidelines 
for  TEC  lesson  validation,  which  is  a  training  effectiveness  evalua¬ 
tion.  The  iterative  process  of  lesson  development  and  the  subsequent 
testing  of  instruction  work  together  to  create,  through  trial  and  re- 


The  training  effectiveness  evaluation,  if  properly  conducted, 
insures  that  the  instruction  teaches;  i.e.,  a  majority  of  the  target 
audience  achieves  the  learning  objectives  at  a  specified  criterion 
level.  If  the  training  objectives  are  properly  determined,  the 
soldiers  will  also  be  able  to  perform  the  required  tasks  at  a  cri¬ 
terion  level.  These  objectives  are  derived  directly  from  the  com¬ 
plete  task  analyses  of  soldier's  jobs.  Nevertheless,  the  TEC  Manager 
is  responsible  for  verifying  the  validity  of  the  training  objectives. 

Subject  Matter  Experts  (SME's)  should,  as  a  minimum,  confirm  the 
validity  of  the  content  of  all  training  objectives.  The  SME  should 
establish  the  validity  of  the  content  by  systematically  checking  that 
the  objectives  have  been  derived  from  an  analysis  of  what  the  soldier 
must  do  in  order  to  perform  the  task  taught  in  the  lesson. 

The  TEC  Manager's  Guidebook  specifies  that  the  training  effective¬ 
ness  of  a  lesson  is  measured  in  terms  of  the  pre-  and  posttest  accom¬ 
panying  it.  The  difference  in  posttest  scores  over  pretest  scores  is 
a  direct  measure  of  how  well  the  lesson  trains.  The  pre-  and  posttest 
are  to  be  validated  to  insure  that  they  are  measuring  what  is  intended 
to  be  measured.  This  reduces  error  in  testing  and  lesson  development, 
and  increases  the  probability  of  effectiveness  of  the  TEC  lesson. 

Pre-  and  Posttest  Development.  The  responsibility  of  a  TEC  lesson 
evaluator  begins  when  training  objectives  are  translated  into  criterion- 
referenced  test  items.  For  example,  the  training  objective — field  strip 
the  Ml 6  rifle  in  total  darkness  within  2  minutes — is  measured  by  cri¬ 
terion-referenced  and  performance  test  items.  These  items  record  the 
GO/NO  GO  of  the  field  stripping  and  the  time  required  to  perform  it. 

In  the  TEC  lesson  development  process,  the  criterion-referenced  test 
items  are  developed  prior  to  the  design  of  the  lesson  itself;  then 
these  test  items  are  incorporated  into  the  pre-  and  posttests.  With¬ 
out  a  valid  test,  there  is  no  way  to  measure  the  training  effectiveness 
of  a  lesson. 

Pre-  and  posttest3  may  be  one  or  a  combination  of  two  types  of  tests 
a  paper-and-pencil  test  or  a  performance  test.  If  a  performance  com¬ 
ponent  is  not  incorporated  into  the  pre-  and  posttests,  a  performance 
test  or  a  separately  validated  written  test  must  be  included  in  addi¬ 
tion  to  the  pre-  and  posttests  for  the  purpose  of  evaluating  training 
effectiveness . 

o  Validity  and  reliability.  Validity  and  reliability 
are  the  two  elements  that  describe  the  utility  and 


accuracy  of  a  test.  Validity  is  the  most  important 
aspect  of  any  type  of  test.  Test  validity  is  defined 
as  the  degree  to  which  a  test  measures  the  performance 
it  was  designed  to  measure.  Test  reliability  is  de¬ 
fined  as  the  consistency  of  the  measure.  This  means 
a  given  test,  no  matter  what  it  is  measuring,  produces 
the  same  value  (score) ,  or  one  very  close,  every  time 
a  person  takes  the  test.  Tests  used  for  TEC  lessons 
must  be  both  reliable  and  valid.  They  must  consistently 
measure  the  effectiveness  of  the  lesson  with  respect 
to  the  soldiers'  real  performance  requirements. 

Developing  a  Valid  Test.  The  validity  of  a  test  can 
only  be  estimated.  But  there  are  known  techniques 
that  can  be  used  in  developing  a  valid  test. 

To  determine  item  content  validity,  the  initial  pool 
of  test  items  should  be  examined  on  an  item-by-item 
basis  to  see  if  each  appears  to  measure  what  it  was 
intended  to  measure.  The  examination  involves  two 
steps:  making  systematic  comparisons  between  training 
objectives  and  the  test  items  designed  to  measure 
these  objectives,  and  having  SME's  review  the  objec¬ 
tives  plus  the  test  items  developed  to  measure  them. 

The  SME  review  should  result  in  revisions  of  the  test 
items  with  respect  to  required  tasks,  applicable  con¬ 
ditions,  and  scoring  standards.  The  review  should 
also  examine  administrative  feasibility  and  define  the 
standardized  testing  conditions.  This  process  should 
produce  content-valid  test  items  which  are  usable  for 
the  test. 

The  test  items  must  then  be  validated  empirically  by 
testing  the  performance  of  two  groups  of  subjects. 

One  group  should  be  experts  (masters)  in  the  perfor¬ 
mance  requirement  the  lesson  is  to  support.  The  other 
group  should  be  novices  (nonmasters) .  The  test  is 
administered  to  both  groups.  The  test  items  which 
discriminate  between  masters  and  nonmasters  are  to 
be  considered  valid  (i.e.,  valid  items  are  those 
which  most  masters  pass  and  most  nonmasters  fail)  . 

The  items  that  do  not  discriminate  should  be  con¬ 
sidered  invalid,  and  eliminated  from  the  test. 

The  procedure  should  be  repeated  using  different 
students  until  an  empirically  valid  test  is  developed. 


o  Establishing  a  Reliable  Test.  There  are  many  methods 
that  can  be  used  to  estimate  a  test's  reliability. 

The  split-half  method  divides  a  test  into  two  parts. 

These  half  tests  are  scored,  correlated,  and  used  in 
an  additional  calculation  to  estimate  the  reliability 
of  the  test. 

The  test  is  administered  as  though  it  had  never  been 
split.  The  test  is  scored  to  produce  two  separate 
scores:  one  for  the  odd-numbered  items  and  one  for 
the  even-numbered  items.  The  scores  are  correlated. 

The  test  is  modified  as  required  until  it  is  both 
valid  and  reliable. 

Individual  Trial  Phase.  An  individual  trial  is  conducted  by  trying 
instructional  materials  individually  on  3  to  5  students  representative 
of  the  target  population.  The  purpose  is  to  identify  those  areas  within 
the  lesson  that  require  revision  to  insure  instructional  effectiveness. 

Small  Group  Trials.  If  no  problems  are  identified  in  the  final  indi¬ 
vidual  trials,  these  trials  may  be  bypassed.  Small  group  trials  are 
conducted  by  simultaneously  trying  instructional  material  on  6  to  10 
students  who  are  representative  of  the  target  population,  but  not  includ¬ 
ed  in  the  individual  trial. 

Effectiveness  Testing  Phase.  The  effectiveness  testing  phase  is  the 
most  important  aspect  in  the  development  of  training  effective  TEC  lessons 
In  this  phase,  it  will  be  determined  whether  the  lesson  is  truly  effec¬ 
tive  with  the  soldiers  for  whom  it  was  designed. 

o  Determining  the  Target  Audience  and  Drawing  a  Sample. 

The  target  audience  consists  of  the  entire  popula¬ 
tion  of  soldiers  to  be  trained  by  the  lesson. 

The  next  step  is  to  draw  a  representative  sample  from 
the  target  population.  The  sample  must  be  representa¬ 
tive  of  the  population  -  not  confined  to  some  peculiar 
subgroup,  such  as  highly  trained  personnel,  or  especially 
selected  personnel  -  so  that  the  results  of  the  testing 
can  apply  to  the  whole  population.  It  is  impractical 
to  draw  randomly  from  the  total  population;  therefore, 
a  sample  must  be  drawn  from  the  subpopulation  available. 

After  identifying  the  subpopulation,  there  are  several 


ways  to  draw  the  actual  sample  of  30  or  more  soldiers 
to  participate  in  lesson  training.  A  simple  and 
practical  method  is  to  obtain  an  ordered  list  of  SSN’s 
(across  units)  and  to  sample  systematically  from  the 
list.  For  example,  if  there  are  500  troops  in  the 
desired  MOS  at  an  installation  and  a  sample  of  35  is 
needed,  an  SSN  listing  for  the  500  troops  should  be 
obtained  and  every  fourteenth  one  selected  for  the 
sample.  This  will  provide  35  names  out  of  the  500. 

Since  some  of  these  people  will  not  be  available  for 
testing,  every  fifteenth  name  is  taken  as  an  alternate 
for  the  preceding  name  on  the  list.  Although  this  is 
not  a  strictly  random  process,  it  will  yield  an  essen¬ 
tially  random  sample  of  the  subpopulation  because  there 
are  normally  no  systematic  biases  in  either  SSN  assign¬ 
ments  or  list  aggregation. 

o  TEC  Lesson  Testing  Methods.  In  TEC  contracts  let  to 
date,  there  are  two  methods  for  testing  TEC  lessons  - 
The  "fixed  sample"  method  and  the  "sequential  testing 
sample"  method. 

When  using  the  fixed  sample  method,  the  number  of  soldiers 
to  be  tested  are  selected  in  advance  (at  least  30) .  They 
are  given  the  pretest,  the  TEC  lesson,  and  the  posttest. 
The  lesson  is  accepted  if,  for  each  training  objective  (TO) 
the  proportion  of  sampled  soldiers  passing  the  posttest 
for  the  TO  exceeds  the  prespecified  criterion  value. 

The  sequential  testing  sample  method  requires  making  a 
number  of  entries  on  a  set  of  charts  (one  for  each  TO) 
after  scoring  each  test.  Based  on  these  entries,  a  de¬ 
cision  is  made  whether  to  accept  the  lesson,  reject  it, 
or  go  on  to  the  next  test.  The  chart  is  discussed  in 
a  subsequent  paragraph. 

o  Pretest  Administration.  Before  administering  the  pretest, 
the  students  should  know  the  purpose  of  the  test  and  the 
lesson  following  it.  During  the  conduct  of  the  test,  the 
testers  must  maintain  an  impartial  attitude.  Students 
passing  the  pretest  cannot  continue  in  the  evaluation 
process.  Feedback  should  not  be  provided  to  students  who 
don't  pass,  because  they  may  learn  the  test  and  not  the 
lesson  objectives. 
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o  Lesson  Administration.  Since  the  goal  is  to  determine 
the  training  effectiveness  of  a  TEC  lesson,  the  test 
situation  should  duplicate  the  real  learning  situation. 

When  a  TEC  lesson  is  ready  for  evaluation  in  a  large 
group  trial,  there  will  usually  be  only  one  copy  available. 
For  this  reason,  most  schools  elect  to  present  the  lesson 
to  the  sampled  soldiers  in  groups  of  five  or  more  at  a 
time,  rather  than  have  each  soldier  review  the  lesson 
individually.  However,  when  a  lesson  is  viewed  in  a  group 
mode,  there  is  little  or  no  self-pacing.  For  certain 
tasks,  the  lack  of  self-pacing  may  decrease  the  effec¬ 
tiveness  of  the  lesson. 

o  Posttest  Administration.  The  posttest  normally  does  not 
differ  from  the  pretest.  After  the  posttest,  each  trainee 
should  be  provided  with  the  results  of  the  test,  and  each 
trainee  should  be  questioned  about  the  quality  and  accept¬ 
ability  of  the  lesson.  Although  trainee  comments  have 
no  bearing  on  the  acceptability  of  the  lesson  in  the 
statistical  sense,  they  should  be  considered  seriously 
and  identified  faults  should  be  corrected. 

Validation  Decision  Making  Phase.  This  phase  of  TEC  effectiveness 
evaluation  is  critical.  In  addition,  it  is  perhaps  the  most  complex 
phase.  This  section  presents  only  the  sequential  sampling  approach  to 
the  validity  decision  process.  With  this  technique,  one  can  feel  con¬ 
fident  that  soldiers  in  the  population  will  be  trained  to  the  selected 
criterion  with  the  least  expenditure  of  manpower  and  other  resources. 

o  Scoring  by  Objective.  All  posttests  are  scored  with 
the  pass-fail  criteria  defined  for  each  objective. 

School  personnel  must  define  the  criteria  beforehand 
and  make  a  scoring  key  for  the  posttest.  Each  soldier's 
responses  on  the  posttest  are  scored  with  this  key. 

After  individual  responses  are  scored,  it  is  determined 
whether  the  soldier  has  passed  or  failed  with  respect 
to  each  training  objective.  The  GOs  and  NO  GOs  for  each 
soldier  are  used  to  determine  whether  the  lesson  is 
training  effective  for  each  objective.  The  scores  are 
entered  on  the  appropriate  charts,  in  the  order  scored, 
to  preserve  the  validity  of  the  process. 


Validation  of  Training  Doctrine. 

The  validation  of  TEC  lessons  should  be  used  as  a  tool  to  verify 
the  accuracy  of  the  doctrine  and  the  effectiveness  of  the  training 
procedures  contained  in  the  lesson.  The  TEC  lesson  should  reflect 
the  best  and  most  current  training  material  available.  If  the  vali¬ 
dation  process  discloses  a  better  training  technique,  or  if  the  TEC 
lesson  is  the  first  major  training  medium  to  be  fielded  after  a  change 
in  training  doctrine,  it  should  contain  that  information. 

The  personnel  responsible  for  TEC  validation  should  be  charged 
with  the  additional  responsibility  of  validation  of  the  training 
doctrine.  If  the  validation  uncovers  problems  concerning  the  vali¬ 
dity  of  the  doctrine,  the  development  process  should  be  halted  until 
doctrinal  questions  are  resolved. 

Lesson  Enhancement. 

The  following  points  should  be  considered  for  their  potential  of 
increasing  TEC  usage: 

o  The  TEC  developer  must  have  a  finite  description 
of  the  target  audience. 

o  The  medium  selected  should  have  a  high  probability 
of  being  used  by  the  target  audience. 

o  The  audiovisual  lessons  should  utilize  the  full 

potential  of  the  medium  -  using  motion  as  necessary 
to  illustrate  the  performance  of  critical  tasks. 

o  The  lesson  should  include  enrichment  information  to 
assist  in  learning  and  understanding  or  to  maintain 
soldier  interest. 

o  The  level  of  complexity  must  be  such  that  the  lesson 
is  not  a  bore  to  the  average  soldier. 

Quantity  vs.  Quality. 

The  number  of  TEC  lessons  produced  by  each  school  should  be  adjusted 
to  insure  that  adequate  resources  (SME's,  hardware,  test  soldiers,  etc.) 
are  available  to  result  in  high  quality  products. 


Hands-on  tests  are  not  being  used  for  hardware-oriented  lessons. 


As  a  rule,  the  validation  process  does  not  include  verification 
of  the  training  doctrine  contained  in  the  lesson.  Additionally, 
there  is  a  general  perception  that  the  TEC  lesson  must  reflect  the 
exact  same  material  located  in  another  official  reference  (FM,  TM, 

SM,  etc . ) . 

Many  lessons  fail  to  utilize  the  full  potential  of  the  audiovisual 
media.  Some  material  is  lifted  directly  from  a  printed  mode.  There 
was  no  motion  in  six  lessons  that  addressed  firing  weapons  and  operat¬ 
ing  vehicles. 

Several  lessons  fail  to  provide  the  enrichment  information  neces¬ 
sary  for  effective  learning  or  full  understanding  of  the  task. 

If  lesson  content  and  support  provided  for  validations  are  valid 
indicators,  the  schools  are  trying  to  develop  more  TEC  lessons  than 
available  resources  will  adequately  support. 

In  general,  the  sample  target  audience  is  not  being  scientifically 
selected,  only  15  soldiers  are  used  for  large  group  validations,  and 
soldiers  that  pass  the  pretest  are  not  eliminated  from  the  validation. 


SUGGESTED  IMPROVEMENTS 


Subject  Matter  Experts. 

The  SME  must  be  actively  involved  in  all  phases  of  lesson  develop¬ 
ment  -  to  include  being  available,  as  required,  on  the  validation  site. 
The  SME  involvement  should  be  of  an  "official"  nature,  in  that  SME 
contributions  to  the  development  effort  are  reviewed  by  the  SME's  super 
visor. 

Hands-on  Tests. 

Hands-on  tests  must  be  included  in  the  validation  to  fully  evaluate 
training  effectiveness. 


CONCLUSIONS 


All  observed  personnel  associated  with  the  TEC  program  were  con¬ 
sidered  to  be  competent  and  dedicated.  The  observed  problems  result 
more  from  their  being  caught  up  in  a  bureaucratic  system  and  inade¬ 
quate  SME  influence  than  from  the  lack  of  individual  effort. 

All  schools  appear  to  be  using  the  TEC  Manager's  Guidebook  as 
a  basis  for  their  TEC  development  efforts.  Personnel  are  familiar 
with  the  guidelines  and  variance  from  guidelines  is  normally  a  func¬ 
tion  of  making  adjustments  considered  necessary  to  accomplish  the 
mission  with  available  resources. 

All  test  soldiers  are  properly  briefed  on  the  purpose  of  the  vali¬ 
dation  and  the  role  they  are  expected  to  perform. 

All  pretests  are  administered  and  scored  in  accordance  with  the 
guidelines.  No  feedback  of  pretest  results  is  being  provided  to 
test  soldiers. 

During  large  group  validations,  some  lessons  are  administered  to 
groups  as  large  as  15;  however,  the  lesson  presentations  appear  to  be 
effective  in  that  all  personnel  have  a  clear  view  of  the  screen  and 
the  duration  of  each  pause  is  adjusted  for  the  slowest  person. 

Posttests  are  administered  following  the  lesson  and  all  results 
are  properly  recorded  on  appropriate  forms. 

Feedback  from  test  soldiers  is  obtained  and  given  consideration  in 
lesson  revisions. 

All  problems  with  the  lesson  or  with  individual  training  objectives 
that  are  discovered  during  the  validation  are  corrected  on  the  spot 
or  recorded  for  subsequent  corrective  action.  This  is  also  true  for 
lessons  that  have  otherwise  been  fully  validated. 

Of  five  schools  and  three  different  contractors  observed,  a  har¬ 
monious  and  cooperative  relationship  appears  to  exist  between  schools 
and  contractors. 

The  SME  does  not  have  sufficient  influence  on  lesson  development. 
The  full  participation  of  the  SME  is  essential  to  the  development  of 
valid  and  effective  lessons. 


Additionally,  previous  experience  indicates  that  the  brighter 
soldiers  are  more  inclined  to  take  advantage  of  off-duty  job  enhance¬ 
ment  aids  such  as  TEC.  Another  factor  to  be  considered  is  the  in¬ 
ability  of  the  Beseler  Cue  See  to  bypass  material.  Even  though  a 
soldier  may  be  fully  knowledgeable  of  a  portion  of  the  lesson,  he 
must  watch  the  entire  lesson  at  normal  speed.  Many  lessons  offer 
little  viewing  incentive  to  the  soldier  that  can  read.  Why  would 
a  soldier  view  the  shot  group  analysis  lesson  for  marksmanship 
training  when  nothing  is  provided  except  what  is  already  clearly 
depicted  on  the  Shot  Group  Analysis  Card? 

Two  lessons  on  transformation  involve  filling  out  two  forms  -  the 
time  required  to  complete  these  lessons  is  estimated  at  4-1/2  hours 
using  a  calculator  and  9  hours  without  a  calculator.  This  is  a  long 
and  complex  but  seldom  used  technique.  No  SQT  would  ever  require 
the  forms  to  be  completed.  The  lessons  provide  little  information 
except  the  guidance  that  is  already  on  the  form.  It  is  the  opinion 
of  the  observer  that  these  lessons  will  seldom  be  used  by  the  target 
audience. 

Enrichment  Information. 

As  in  the  above  examples,  most  TEC  lessons  observed  failed  to 
provide  a  hint  of  why  a  procedure  is  performed  or  its  relative  impor¬ 
tance  to  other  procedures.  This  type  of  information  would  assist  in 
understanding  that  it  is  more  serious  to  run  an  engine  without  oil  or 
water  than  with  a  tachometer  that  is  50  RPM  too  high  at  idle,  and  that 
trigger  squeeze  is  more  important  to  marksmanship  than  the  precise 
placement  of  the  finger  on  the  trigger.  Additionally,  this  informa¬ 
tion  could  eliminate  confusion:  Why  is  the  gas  turned  on  prior  to 
cranking  the  Vulcan  Carrier  in  the  summer  and  turned  off  prior  to 
cranking  in  the  winter? 

The  Validation  Requirement. 

TEC  validation  appears  to  be  old  hat  and  has  fallen  into  the  realm 
of  something  else  that  has  to  be  accomplished.  A  standard  format/pro¬ 
cedure,  based  on  TRADOC  guidelines,  is  used;  however,  there  seems  to  be 
no  rigid  adherence  to  any  policy.  Changes  are  freely  made  to  accom¬ 
modate  j.ack  of  troop  support,  lack  of  equipment,  lack  of  knowledge,  and 
to  speed  up  the  validation  process.  The  schools  should  make  a  decision 
on  validation  requirements  for  each  lesson.  A  good  review  by  a  know¬ 
ledgeable  subject  matter  expert  may  contribute  more  to  a  valid  lesson 
than  a  go-through-the-motions  validation. 
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Trainer/Developer  Interface. 


During  the  development  and  validation  of  a  TEC  lesson,  the 
lesson  developer  gets  very  familiar  with  many  training  requirements 
that  could  be  simplified  through  a  minor  change  or  modification  to 
a  specification  or  item  of  hardware. 

Observation  of  TEC  lessons  reveals  that  training  requirements 
have  not  been  given  adequate  consideration  by  technical  personnel. 

The  training  requirements  in  the  lessons  are  presented  exactly  as  the 
information  comes  from  a  technical  manual.  For  example,  in  one  lesson 
the  soldier  is  taught  to  remember  that  it  is  very  important  for  the 
tachometer  of  a  certain  vehicle  not  to  exceed  2975  RPM.  It  is  impos¬ 
sible  to  read  2975  RPM  on  the  tachometer  even  when  the  vehicle  is  not 
moving,  while  a  reading  of  3000  can  be  read  accurately  with  ease. 

The  additional  25  RPM  of  engine  speed  would  be  negligible  on  the  life 
of  the  engine  and  drive  train.  In  the  same  lesson,  the  soldier  is 
taught  that  if  the  temperature  is  above  40°  F,  he  should  warm  the 
engine  for  3  to  5  minutes  at  800  to  1000  RPM  and  then  idle  at  550  to 
600  RPM.  If  the  temperature  is  below  40°  F,  warm  the  engine  for 
3  to  5  minutes  at  550  to  1000  RPM  and  then  do  a  second  warm-up  for 
5  minutes  at  1200  to  1500  RPM  and  then  warm  the  transmission  for 
3  to  10  minutes  at  800  to  1000  RPM.  The  soldier  is  also  required  to 
memorize  such  things  as  temperature,  oil  pressure,  and  air  pressure 
readings.  Most  of  these  readings  must  be  taken  from  gauges  that  give 
no  indication  of  minimum  or  maximum  operating  limits.  A  simple  color 
code  on  the  gauges  would  greatly  enhance  training.  It  appears  that 
the  complexity  of  the  training  could  be  significantly  reduced  if  some 
procedure  was  available  to  get  the  technician  and  trainer  together. 

The  Target  Audience. 

During  these  visits,  several  soldiers  were  heard  to  complain  that 
a  particular  TEC  lesson  was  an  insult  to  their  intelligence.  A  major 
objective  of  the  validation  is  to  insure  that  the  lesson  teaches  the 
target  audience.  In  an  effort  to  produce  a  good  product,  some  very 
high  validation  criteria  have  been  established.  If  90%  of  the  target 
audience  is  required  to  respond  correctly  to  90%  of  all  items,  it 
logically  follows  that  the  difficulty  level  and  complexity  of  each 
lesson  must  be  adjusted  to  allow  soldiers  with  the  lowest  intellectual 
capacity  to  achieve  a  90%  performance  after  one  viewing.  There  should 
be  an  awareness  that  the  simplicity  of  many  lessons  appears  to  be  a 
turn-off  for  some  soldiers. 


of  the  lesson,  and  there  should  also  be  enough  flexibility  to  make 
changes  necessary  for  the  most  effective  presentation  of  the  material 
in  an  audiovisual  mode. 

The  personnel  involved  with  TEC  validation  should  be  charged  with 
the  validation  of  the  doctrine  and  training  procedures  contained  in  the 
lesson  as  well  as  the  effective  teaching  of  training  objectives. 

Subject  Matter  Expertise. 

The  purpose  of  this  effort  was  to  evaluate  the  procedures  being 
used  to  validate  TEC  lessons.  Except  for  the  lack  of  hands-on  perfor¬ 
mance  tests,  no  major  problems  were  evident.  However,  several  questions 
are  raised  concerning  the  validity  of  the  doctrine/training  procedures 
contained  in  the  TEC  lessons. 

The  "system"  has  resulted  in  non-subject-matter-experts  being  assign 
ed  the  primary  responsibility  of  producing  a  large  number  of  TEC  lessons 
which  cover  a  broad  range  of  subject  areas.  The  constraints  of  time, 
the  lack  of  validation  personnel  who  meet  minimum  prerequisites,  the 
scarcity  of  hardware,  subject-matter  experts  and  test  facilities,  con¬ 
tribute  to  a  "cut  the  corners"  approach  to  TEC  development. 

There  are  no  subject  matter  experts  (SME)  directly  responsible  to 
the  personnel  charged  with  the  development  of  the  TEC  lessons.  The 
SMEs  perform  this  function  as  an  additional  duty  and  seldom  have  major 
input  when  the  paper  is  blank.  The  lack  of  adequate  SME  influence 
and  ill-conceived  doctrine,  coupled  with  improper  validation  procedures, 
result  in  some  products  that  appear  less  than  satisfactory. 

Working  under  the  assumption  that  the  effectiveness  of  the  valida¬ 
tion  process  may  be  judged  by  an  evaluation  of  the  product  produced  by 
the  process,  three  of  the  audiovisual  lessons  were  informally  examined 
to  determine  the  technical  accuracy  of  the  doctrine  presented  and  to 
determine  what  contribution  the  lesson  content  may  have  to  the  actual 
performance  of  the  task  in  the  real  world. 

All  lessons  observed  were  in  the  final  stages  of  development;  there¬ 
fore,  it  is  doubtful  that  major  changes  have  been  made  to  lesson  content 
The  critique  of  selected  lessons  at  Appendix  B  is  intended  to  show  the 
lack  of  adequate  SME  influence  and  to  support  previous  and  subsequent 
comments . 


focused  more  on  the  short  term  memory  capability  of  test  soldiers 
than  on  the  validity  or  accuracy  of  the  doctrine  or  training  proce¬ 
dure  contained  in  the  lesson.  Also,  since  there  were  no  hands-on 
performance  tests  for  hardware-oriented  lessons  anywhere  in  the  de¬ 
velopment  cycle,  there  was  no  assurance  that  the  lesson  actually 
presented  the  material  necessary  to  perform  the  required  tasks. 

But  what  if  the  material  comes  directly  from  a  Field  Manual?  Most 
schools  operate  under  the  policy  that  all  material  contained  in  a  TEC 
lesson  must  be  supported  by  a  current  Field  Manual  (or  similar  author¬ 
ity)  or  a  final  approved  draft.  This  policy  has  the  obvious  advantage 
of  keeping  everything  uncomplicated  in  that  the  TEC  lesson  tracks  with 
the  FM,  Soldier's  Manual,  SQT,  etc.  Since  the  TEC  lesson  can't  vary 
from  current  doctrine,  the  TEC  developer  and  contractor  should  have 
minimal  changes  to  their  contract  time  schedule.  Additionally,  since 
the  developers  of  TEC  lessons  don't  claim  to  be  subject  matter  experts, 
it  makes  their  job  easier  because  they  don't  have  to  vary  from  what  is 
in  the  "book."  However,  this  policy  tends  to  focus  attention  on  the 
validation  of  the  lesson  as  written  while  avoiding  any  validation  of 
the  doctrine  or  training  procedures  being  addressed.  More  time  and 
money  is  expended  in  the  validation  of  some  TEC  lessons  than  in  the 
validation  of  the  original  doctrine  or  training  procedure  upon  which 
the  lesson  is  based.  Therefore,  in  effect,  time  and  effort  is  expend¬ 
ed  to  validate  the  training  of  unvalidated  doctrine.  Unfortunately, 
the  TEC  lesson  will  normally  validate  because  of  the  validation  pro¬ 
cedures  used.  It  appears  the  Army  has  much  to  gain  if  TEC  validations 
could  be  used  to  verify  current  doctrine  and  the  most  effective  train¬ 
ing  procedures. 

Changes  to  Army  training  literature/media  are  slow.  When  a  change 
is  made,  it  normally  takes  years  to  reflect  that  change  in  Field  Manuals 
Soldier's  Manuals,  TEC  lessons,  etc.  TEC  lessons  should  contain  the 
best  and  most  current  doctrine  and  training  procedures.  The  TEC  lessons 
produced  today  should  provide  maximum  assistance  to  the  soldier  in  pro¬ 
perly  and  effectively  performing  his  job.  The  requirement  that  all  TEC 
material  be  contained  in  a  Field  Manual  that  may  be  several  years  old 
and  can ' t  be  revised  for  two  years  does  nothing  to  enhance  the  TEC 
lesson. 

It  appears  that  the  content  of  many  TEC  lessons  is  transferred  from 
a  Field  Manual  to  an  audiovisual  format  by  a  non-subject  matter  expert. 

A  subject  matter  expert  should  be  involved  in  the  lesson  development 
(not  just  as  a  consultant)  to  insure  the  most  current  doctrine  is  re¬ 
flected,  to  provide  correct  emphasis  to  the  critical  teaching  points 


pre-  and  posttest  were  not  treated  as  firm  testing  criteria.  When 
test  soldiers  had  difficulty  with  a  posttest  question,  for  about  one- 
half  of  the  cases  observed,  corrective  action  was  directed  at  changing 
the  question  rather  than  changing  lesson  content  to  better  teach  the 
task.  All  pre-  and  posttests  for  hardware-oriented  lessons  were  of 
paper-and-pencil  type — no  hands-on  testing.  It  is  difficult  to  ade¬ 
quately  evaluate  training  effectiveness  without  using  a  hands-on-the- 
equipment  performance  test.  Additionally,  in  no  case  was  the  test 
itself  validated  prior  to  use  in  the  lesson  validation. 

The  Memory  Game. 

In  all  cases,  the  pre-  and  posttest  were  the  same  and,  in  most 
cases,  the  pre/posttest  was  nearly  identical  to  the  self-evaluation 
contained  in  the  lesson.  Sometimes  the  entire  validation  is  accomplish¬ 
ed  in  one  hour.  This  includes  the  pretest,  the  lesson  which  includes 
a  review  of  each  question  and  the  self-evaluation,  and  the  posttest. 

Since  a  hands-on  test  was  not  required,  the  obvious  question  of  rote 
memory  must  be  given  some  consideration.  Of  all  TEC  validations  ob¬ 
served,  only  one  pretest/posttest  question  was  noted  as  being  different 
from  the  self-evaluation  questions  contained  in  the  lesson.  This  in¬ 
volved  an  instrument  panel  with  six  gauges.  When  used  in  the  pretest, 
all  six  gauges  were  shown  with  readings  that  were  within  the  normal 
operating  range  for  the  particular  vehicle.  As  normal,  no  feedback 
of  pretest  results  was  provided.  During  the  lesson,  instrument  panels 
were  shown  several  times,  but  each  time  at  least  one  gauge  reflected 
an  improper  reading.  The  correct/incorrect  reading  of  each  gauge  was 
clearly  presented  and  each  gauge  was  shown  several  times  with  a  correct 
reading;  however,  every  soldier  missed  the  posttest  question  that  showed 
an  instrument  panel  with  all  gauges  correct.  A  retention  test,  following 
initial  lesson  exposure  by  only  a  few  days  or  a  few  hours,  may  add  sig¬ 
nificantly  to  the  validity  of  posttesting. 

What  is  Validated? 

This  question  concerns  the  responsibility  of  personnel  developing 
TEC  material  to  verify  or  confirm  the  validity  of  the  doctrine  or  train¬ 
ing  procedure  contained  in  the  TEC  lesson.  As  an  example,  the  TEC 
writers  may  develop  an  item  which  is  incorrect  and  a  newly  assigned  SME 
may  concur  with  the  item.  This  is  presented  to  test  troops  (questions 
and  answers)  four  times  in  a  short  time  interval  and  then  they  are  asked 
the  question  on  the  posttest.  If  90%  of  the  test  troops  have  a  good 
memory,  or  if  they  are  allowed  to  keep  the  answers  they  wrote  during  the 
lesson,  the  item  will  validate.  In  effect,  the  TEC  validations  observed 


Lesson  Administration. 


Three  of  the  audiovisual  lessons  were  administered  by  projecting 
35mm  slides  on  a  large  screen  and  coordinating  it  with  audio.  The 
duration  of  each  pause  was  determined  by  the  slowest  test  soldier. 

The  Armor  School  used  the  Singer  Caramate  II  with  four  soldiers  sit¬ 
ting  directly  in  front  of  the  small  screen.  The  printed  test  lessons 
were  administered  on  an  individual  basis,  each  soldier  moving  through 
the  lessons  at  their  own  pace . 

Posttest  Administration. 

All  posttests  were  conducted  immediately  following  the  lesson. 
Normally,  a  few  minutes  were  provided  for  the  test  soldiers  to  review 
notes  taken  during  the  lesson  and  the  test  was  administered  without 
the  use  of  notes.  In  one  case,  the  soldiers  were  allowed  to  retain 
all  notes  taken  during  the  lesson.  No  equipment  or  hands-on  perfor¬ 
mance  tests  were  included  in  the  validation  of  hardware-oriented  lessons 

Lesson  Validation. 

Each  school  had  somewhat  different  criteria  for  the  lessons  under¬ 
going  validation.  The  pass  rate  required  for  a  lesson  to  validate 
ranged  from  70  to  100  percent,  based  on  the  importance  rating  assigned 
by  the  schools  to  a  particular  task  or  group  of  tasks.  All  posttest 
results  were  properly  recorded  on  the  appropriate  forms. 

Responsibility  for  Validations. 

The  contractor  is  technically  responsible  for  the  conduct  of  the 
validation,  but,  in  practice,  it  appears  to  be  a  coordinated  effort 
between  the  schools  and  the  contractors. 


DISCUSSION 


Pre-  and  Posttest  Development. 

Without  a  valid  posttest  to  measure  the  training  effectiveness  of 
a  lesson,  there  is  no  way  to  determine  if  the  lesson  is  effective. 
Criterion  referenced  test  items  should  be  developed  prior  to  the  de¬ 
sign  of  the  lesson.  The  impression  from  these  visits  was  that  the 


OBSERVATIONS 


Sample  Target  Audience. 

All  lessons  observed  were  undergoing  large  group  validation.  The 
actual  number  of  soldiers  utilized  for  these  validations  varied  from 
six  to  thirty,  with  approximately  fifteen  being  the  normal  objective. 
Four  of  the  five  schools  stated  that  fifteen  was  the  maximum  number 
of  soldiers  that  could  realistically  be  obtained  for  the  purpose  of 
TEC  validation.  Some  schools  conduct  all  validations  at  the  home 
installation,  while  others  conduct  validations  both  at  home  and  in 
field  locations.  In  all  cases,  the  constraints  of  time,  personnel, 
money,  and  equipment  received  primary  consideration  in  these  decisions. 
The  method  of  selecting  the  sample  target  audience  was  normally  by 
rank  and  MOS,  with  this  commitment  often  being  part  of  the  post-wide 
detail  requirements.  Some  soldiers  were  over-qualified,  some  had  an 
MOS  that  did  not  relate  to  the  lesson,  and  some  were  "professional 
TEC  validators"  in  that  they  would  volunteer  every  time  their  unit 
had  a  requirement  to  provide  soldiers  for  the  purpose  of  TEC  valida¬ 
tion.  Part  of  the  problem  associated  with  the  sample  target  audience 
appears  to  be  linked  to  something  less  than  a  clear  understanding  of 
precisely  who  would  use  the  lesson  when  it  was  fielded. 

Testing  Methods. 

The  fixed  sample  method  of  testing  was  used  by  four  of  the  five 
schools.  The  Armor  School  used  a  modification  of  the  sequential  test¬ 
ing  method  -  30  soldiers  were  scheduled  for  the  validation,  but  the 
lesson  was  administered  to  only  four  soldiers  at  a  time.  Most  schools 
used  the  sequential  charts  for  recording  test  results . 

Pretest  Administration. 

Ml  pretests  were  generally  conducted  in  the  same  manner  -  the  pre¬ 
test  was  administered,  scores  were  recorded,  and  no  feedback  was  pro¬ 
vided  to  test  personnel.  In  all  cases,  the  pretest  and  posttest  were 
exactly  the  same.  Criteria  were  established  for  the  elimination  of 
the  personnel  based  on  a  high  level  of  performance  on  the  pretest; 
however,  no  personnel  were  eliminated.  In  one  case,  60%  of  the  sample 
target  audience  passed  the  pretest,  and  in  another,  virtually  all  would 
have  passed  if  they  had  been  given  adequate  time  and  motivation  to  com¬ 
plete  the  pretest. 


If  the  lesson  satisfactorily  meets  the  criteria  on 
all  training  objectives  (and  if  there  are  no  other 
indications  of  quality  or  acceptance  problems) ,  the 
lesson  will  pass  on  to  the  final  stages  of  develop¬ 
ment,  production,  and  distribution.  If  the  lesson 
achieves  the  criteria  with  respect  to  most  of  the 
objectives,  it  should  probably  be  revised  and  re¬ 
tested.  Scrap  only  those  lessons  which  totally 
fail  to  train  on  the  major  objectives.  Revise  any 
lesson  which  fails  to  validate  with  respect  to  any 
training  objective.  This  process  must  be  performed 
fully  and  accurately  to  insure  that  soldiers  in  the 
field  get  a  "certified  training  effective"  lesson. 


PURPOSE 


To  determine  the  current  practices  and  procedures  used  by  the 
service  schools  to  validate  TEC  lessons  and  to  evaluate  the  effective¬ 
ness  of  validation  efforts. 


PROCEDURE 


During  the  period  Aiigust  to  November  1978,  TEC  validations  being 
conducted  by  five  different  service  schools  were  observed  by  personnel 
of  Litton-Mellonics  and  the  Army  Research  Institute.  The  table  at 
Appendix  A  provides  data  concerning  each  visit.  In  general,  these  were 
informal  visits  conducted  for  the  purpose  of  observing  TEC  validation 
procedures  and  formal  feedback  was  not  provided  to  the  schools  concern¬ 
ing  the  findings  of  these  visits. 

Three  schools  were  validating  hardware-oriented  audiovisual  lessons 
(Infantry,  Armor,  and  Air  Defense) .  The  Artillery  School  was  validat¬ 
ing  audiovisual  lessons  which  involved  filling  out  forms,  and  the  Signal 
School  was  validating  several  printed  text  lessons  which  focused  on  the 
handling  of  messages  in  a  communications  center.  All  general  points 
discussed  are  based  on  observations  and  impressions  at  two  or  more 
locations . 


NUMBER  OF  FAILURES 


TEST  DATA: 


STUDENT: 


STUDENT: 


GO 

GO 

NO  GO 
GO 
GO 
GO 

NO  GO 


HIGH  75% 

DESIRABLE  65% 
LOW  55% 

O' =.05  J3  =  .25 


LESSON  NO.  _ 

KIT  TITLE  _ 

OBJECTIVE  NO.  _ 

CERTIFICATION  _ 
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NUM8EB  OF  TRIALS 


Figure  1 :  Sequential  Testing  Chart 


Acceptance/Rejection  of  TEC  Lesson  by  Objective. 

When  the  above  procedures  are  completed,  the  charts 
will  reflect  whether  the  training  objectives  are 
successful  or  unsuccessful.  Then  a  decision  must 
be  made  with  respect  to  the  total  lesson.  The  follow 
ing  questions  should  be  asked:  Is  the  lesson  effec¬ 
tive  enough  to  be  sent  forward  for  final  development? 
Is  it  worth  revising  where  it  is  weak?  Or  is  it  so 
poor  that  it  should  be  scrapped? 


o  o  o  o  o  o  o 
zuuooou 


Selecting  Charts  for  Each  Training  Objective.  The 
TEC  contracts  include  five  distinct  charts  for  the 
sequential  scoring  process.  These  charts  are  used 
to  evaluate  the  training  effectiveness  of  the  TEC 
lesson  with  respect  to  each  objective.  The  chart 
to  be  used  for  each  objective  is  determined  by  the 
importance  rating  assigned  to  that  objective. 

Plotting  GO/NO  GO  Scores.  The  sequential  testing 
charts  are  graphs  on  which  the  GO/NO  GO  scores  of 
each  soldier  are  plotted  with  respect  to  an  indi¬ 
vidual  objective.  Five  numbers  are  given  on  each 
chart.  Three  of  the  numbers  represent  a  "high," 
"desirable,"  and  "low"  value  for  the  percent  of 
soldiers  receiving  a  GO  for  a  given  training  ob¬ 
jective.  Two  of  the  numbers  axe  called  alpha  (a) 
and  beta  (b) .  These  numbers  are  interpreted  as 
follows : 

"a"  is  the  probability  that  an  objective  will 
be  rejected  if  the  percent  of  soldiers  in  the 
target  population  who  would  receive  GO  for  that 
objective,  if  they  viewed  the.  lesson,  is  equal  to 
the  "high"  value. 

"b"  is  the  probability  that  an  objective  will  be 
accepted  if  the  percent  of  the  soldiers  in  the 
target  population  who  would  receive  GO  for  that 
objective,  if  they  viewed  the  lesson,  is  equal 
to  the  "low"  value. 

The  horizontal  axis  counts  the  number  of  trials 
(soldiers  taking  the  test)  and  the  vertical  axis 
counts  the  cumulative  number  of  NO  GOs.  The  graph 
is  simply  a  mechanical  way  to  determine  whether  the 
lesson  should  be  accepted  or  rejected,  based  on  the 
results  of  the  tests,  for  that  objective.  The 
parallel  lines  on  the  chart  mark  the  regions  for 
acceptance  or  rejection. 


Usually  the  result  of  a  test  is  a  mixture  of  GOs 
and  NO  GOs,  rather  than  a  situation  that  results 
in  all  GOs  or  NO  GOs.  In  this  example  (see  Figure  1) 
it  takes  14  trials  to  reach  the  accepted  area. 
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APPENDIX  B 


CRITIQUE  OF  SELECTED  TEC  LESSONS 


These  comments  are  based  on  the  premise  that  the  effectiveness 
of  TEC  lesson  review  and  validation  procedures  can  be  judged  by  a 
critical  examination  of  the  completed  TEC  lesson.  The  comments  con¬ 
cerning  technical  accuracy  of  lesson  content  are  based  on  the  opinion 
of  the  author  and  some  points  may  be  questionable.  However,  it  is 
believed  that  this  review  accomplishes  the  intended  purpose  of  pro¬ 
viding  strong  evidence  that  lesson  content  leaves  something  to  be 
desired. 


Zeroing  the  M16A1  Rifle  -  Analyzing  and  Correcting  Errors. 

The  soldier  is  directed  to  have  the  Rifle  Shot  Group  Analysis  Card 
(GTA  21-1-4)  in  hand  when  this  lesson  is  viewed.  The  problems  with 
this  lesson  begin  with  the  card  (Figure  2) ,  which  is  filled  with  faulty 
information.  Sight  alinement  is  presented  as  the  most  important  factor 
in  shooting  the  M-16.  The  sight  alinement  sketch  that  shows  the  path 
of  the  bullet  if  an  error  is  made  in  sight  alinement  is  reproduced  in 
one  of  the  lesson  visuals.  Emphasis  is  placed  on  sight  alinement 
several  times  as  being  "most  critical,"  bullets  will  be  "way  off," 

"won't  be  near  the  target,"  "lucky  if  he  found  any  holes,"  etc.  The 
infatuation  with  sight  alinement  seems  to  stem  from  the  fact  that  the 
error  increases  with  range.  It  is  impressive.  The  sight  radius  on 
the  M-16  is  50.17  centimeters  or  about  1/2  meter.  This  means  that 
any  error  made  in  the  alinement  of  the  front  and  rear  sights  will  be 
multiplied  50  times  at  25  meters,  200  times  at  100  meters,  and  600  times 
at  300  meters.  However,  the  sketch  shows  the  front  sight  post  about 
1/4  of  the  distance  between  the  center  and  the  inside  edge  of  the  rear 
sight.  The  rear  sight  aperture  is  2  millimeters  from  side  to  side  - 
1  millimeter  from  the  center.  Therefore,  the  sight  misalinement  de¬ 
picted,  .25  millimeter,  times  600,  is  equal  to  150  millimeters  or 
something  less  than  6  inches.  This  means  that  the  19-1/4  inch  silhou¬ 
ette  would  be  hit  at  300  meters  -  not  missed  by  40  inches.  The  accu¬ 
racy  of  this  information  has  been  confirmed  by  several  live  firing 
tests;  however,  it  can  be  verified  by  spending  a  few  minutes  on  any 
range . 

The  top  two  shot  groups  on  this  card,  long  vertical  and  long  hori¬ 
zontal,  are  incorrect  for  four  reasons: 


IF  YOUR  SHOT  GROUP 
LOOKS  LIKE  THIS 

LONG  VERTICAL 


LONG  HORIZONTAL 


SHORT  VERTICAL 


SHORT  HORIZONTAL 


LONG  VERTICAL: 
IMPROPER  BREATHING 


LINE  OF 
SIGHT 


LOW  RIGHT  : 

IMPROPER  TRIGGER  CONTROL 
LEFT  ELBOW  SLIPPING  OUT 


SCATTERED  : 

RE-EXAMINE  ALL  STEADY 
HOLD  FACTORS 


PATH  OF\ 
BULLET 


ERROR  IN  SIGHT 
ALINEmENT. 
ERROR  INCREASES 
as  Range  increases 


LINE 

OF 

SIGHT. 
PATH  ' 
OF 

BULLET 

ARE 

SAME 


ERROR  IN  PLACEMENT 
OF  aiming  POINT. 

ERROR  DOES  NOT 
INCREASE  AS  Range 
INCREASES 


THIS  INDICATES  THAT  SIGHT 
ALINEmENT  IS  MORE  IMPORTANT 
THAN  AIMING  POINT 


SCORING  TEMPLATE 
USED  TO  CHECK 
SHOT  GROUP  ON 
target 

(not  TO  SCALE) 


SEMIAUTOMATIC  RIFLE  SHOT  GROUP  ANALYSIS 
CIRCLE  "A"  -3c™  DIAMETER: 

SATISFACTORY  3-ROUND  SHOT  GROUPS  FIRED  FROM  THE  PRONE  SUPPORTED 
OR  FOXHOLE  POSITION  AT  25METER5  MUST  FALL  WITHIN  OR  TOUCH  THIS  CIRCLE. 
CIRCLE  DIAMETER  : 

SATISFACTORY  3 -round  snot  groups  fired  from  the  kneeling  supported 

AND  ALL  UNSUPPORTED  POSITIONS  AT  25  METERS  MUST  FALL  WITHIN  OR  TOUCH 
THIS  CIRCLE. 


o  Per  the  above  discussion. 


o  It  is  obvious  that  the  analysis  is  made  based  on 
an  assumption  that  the  rifle  is  zeroed;  therefore, 
all  bullet  strikes  should  be  shown  2.4  centimeters 
below  point  of  aim  -  not  the  same  as  point  of  aim. 

o  The  alinement  error  depicted  is  more  than  an  indi¬ 
vidual  can  unknowingly  make.  If  the  sights  are 
misalined  as  shown  for  "long  horizontal  A,"  the 
rear  sight  aperture  will  be  darkened  to  such  an 
extent  that  this  could  not  logically  be  an  unin¬ 
tentional  error.  A  look  through  the  sight  of  an 
M-16  rifle  will  verify  this  point. 

o  The  average  M-16  rifle  with  service  ammunition  is 
only  capable  of  firing  shot  groups  with  a  spread  of 
1/2  inch  to  one  inch  at  25  meters .  The  firer  may 
be  expected  to  add  another  1/2  inch  to  this  probable 
error.  Therefore,  each  bullet  fired  may  strike  3/4 
of  an  inch,  in  any  direction,  from  the  point  of  aim  - 
all  other  sources  of  error  at  a  minimum.  It  seems 
absurd  to  assume  that  sight  alinement  error  will  be 
either  perfectly  vertical  or  perfectly  horizontal 
and  even  more  absurd  to  assume  that  these  perfectly 
vertical  or  horizontal  errors  will  result  in  per¬ 
fectly  vertical  or  horizontal  placement  of  shots. 

The  other  five  shot  groups  depicted  on  the  card  are 
as  erroneous  as  the  two  discussed  above. 

The  target  in  the  lesson  has  squares  of  1.4  centimeters  and  a  5.2 
centimeter  zeroing  template  is  used .  By  the  time  this  lesson  is 
fielded,  it  is  expected  that  a  target  with  .7  centimeter  squares 
and  a  4  centimeter  zeroing  circle  will  be  in  standard  use. 

In  effect,  this  lesson  validated  because  the  soldierr  vere  capable 
of  looking  at  a  placement  of  shots  on  the  shot  group  analysis  card  and 
relating  that  to  the  exact  same  placement  of  shots  on  the  posttest. 

A  soldier  completing  this  lesson  should  be  capable  of  analyzing  a  shot 
group  and  identifying  the  probable  causa  of  unsatisfactory  groups. 
There  is  no  indication  that  personnel  viewing  this  lesson  can  analyze 
real  shot  groups. 

Failure  to  question  material  contained  in  a  Field  Manual,  coupled 
with  the  failure  to  conduct  hands-on  testing,  has  resulted  in  the  per¬ 
petuation  of  erroneous  training  doctrine. 


Zeroing  the  M16A1  Rifle  -  Aiming  and  Firing  Techniques 


The  primary  problem  with  this  lesson  is  that  emphasis  is  placed 
on  the  wrong  points. 

The  use  of  the  eight  steady  hold  factors  to  teach  firing  tech¬ 
niques  is  a  questionable  procedure,  since  few  of  the  eight  factors 
can  be  directly  related  to  getting  the  bullet  on  target.  However,  a 
discussion  of  that  idea  would  be  beyond  the  scope  of  this  paper.  In 
any  case,  some  factors  are  much  more  important  than  others  in  obtain¬ 
ing  an  effective  zero. 

Trigger  control  is  the  most  important  of  the  eight  steady  hold 
factors  -  trigger  squeeze  being  the  key  to  good  shot  groups .  Rear¬ 
ward  trigger  pressure  is  mentioned  twice  with  the  same  emphasis  as 
40  other  items  while  placing  too  much  finger  on  the  trigger  receives 
major  emphasis.  The  point  is  made  at  least  four  times,  reinforced 
with  visuals,  that  only  the  tip  of  the  finger  must  be  on  the  trigger. 

In  fact,  most  M-16  rifles  require  7-10  lbs.  of  pressure  to  pull  the 
trigger,  and  for  many  people,  putting  more  than  the  finger  tip  on 
the  trigger  improves  their  firing  performance. 

The  other  point  that  receives  major  emphasis  in  the  lesson  is  the 
placing  of  the  rifle  directly  on  the  sandbag  support  -  the  lesson 
states  that  this  is  wrong,  that  it  should  never  be  done.  The  basic 
fundamental  of  getting  a  good  zero  is  to  hold  the  weapon  in  a  steady 
position;  therefore,  the  full  utilization  of  all  available  support 
should  be  encouraged.  There  is  nothing  wrong  with  placing  the  weapon 
directly  on  the  support  during  the  zero  process.  An  argument  could 
be  made  for  this  being  a  better  procedure;  however,  the  point  is  that 
it  is  clearly  not  one  of  the  40  items  covered  in  the  lesson  that  should 
be  selected  to  receive  special  emphasis. 

This  lesson  clearly  indicates  the  lack  of  adequate  SME  influence. 
Additionally,  a  brief  firing  exercise  would  demonstrate  that  the  points 
receiving  major  emphasis  in  the  lesson  have  no  positive  influence  on 
the  strike  of  the  bullet. 


During  Operation  Maintenance  Checks  on  the  Five-Ton  Truck. 

The  organization  of  this  material  is  questionable.  The  objective 
is  to  present  before,  during,  and  after  operation  maintenance  checks 
for  the  M809  and  M39  trucks,  and  the  material  is  organized  into  three 
lessons  -  before,  during,  and  after  checks  for  the  M809  and  M39. 


However,  there  are  numerous  differences  between  the  two  vehicles 
that  the  student  is  asked  to  put  to  memory  (idle  speed,  temp,  oil 
pressure,  air  pressure,  etc.).  A  more  logical  organization  appears 
to  be  two  lessons,  each  covering  before,  during,  and  after  checks 
for  each  series  of  vehicles. 

The  lesson  appears  to  be  about  two  trucks,  the  M809  and  the  M39. 

The  technical  manual  (TM)  indicates  that  there  are  20  different  models 
of  the  M809  series  and  12  different  models  of  the  M39  series  (cargo, 
wrecker,  dump,  van,  etc.) .  The  maintenance  checks  discussed  in  this 
lesson  apply  to  all  32  models.  It  seems  appropriate  to  mention  this 
and  to  show  some  pictures.  The  lesson  currently  makes  no  visual  dis¬ 
tinction  between  the  M809  and  M39 . 

The  lesson  states  that  the  M39  may  be  Diesel  or  Multifuel.  Accord¬ 
ing  to  the  TM,  it  may  be  Diesel,  Multifuel,  or  Gas.  The  lesson  makes 
reference  several  times  to  the  M39  Multifuel  when  it  should  just  refer¬ 
ence  the  M39. 

The  lesson  requires  the  soldier  to  memorize  the  idle  speed  of  700  RPM 
In  fact,  the  idle  speed  on  these  vehicles  ranges  from  400  to  800  RPM. 

The  lesson  currently  directs  the  soldier  to  check  the  idle  for  his  par¬ 
ticular  vehicle,  but  it  seems  useless  to  remember  700  since  that  will 
not  apply  to  any  vehicle.  The  lesson  teaches  that  if  the  idle  is  not 
at  the  designated  setting,  the  driver  should  cease  operations,  stop 
the  motor,  and  report  this  to  organizational  maintenance.  In  fact, 
the  idle  adjustment  is  a  minor  maintenance  problem.  It  should  be  noted 
on  the  Form  2404  and  reported  to  maintenance  personnel,  but  the  driver 
should  not  cease  operations  while  on  a  mission. 

The  lesson  teaches  that  if  the  tachometer  is  in  the  red,  the  driver 
should  cease  operations,  stop  the  motor,  and  report  to  organizational 
maintenance  that  the  engine  is  racing.  This  is  like  telling  a  traffic 
cop  that  you  were  speeding  because  the  gas  pedal  was  mashed  down  too 
far.  The  engine  is  racing  because  the  driver  has  the  pedal  to  the  metal. 
This  condition  is  corrected  by  lifting  the  right  foot  or  changing  to 
a  higher  gear. 

The  lesson  teaches  that  the  oil  pressure  at  35  MPH  is  25-70  lbs. 
for  the  M809  and  50-65  lbs.  for  the  M39.  In  fact,  this  is  the  normal 
range  for  highway  driving  and  a  constant  35  MPH  pressure  would  be  in 
a  more  narrow  range. 


The  lesson  teaches  that  if  the  oil  pressure  gauge  is  not  correct 
and  the  oil  level  is  correct,  the  driver  should  report  that  he  has 
a  faulty  oil  pressure  gauge.  This  is  incorrect.  It  is  not  a  safe 
bet  that  the  gauge  is  bad.  The  driver  should  report  that  his  oil 
pressure  reading  is  incorrect. 

The  lesson  teaches  that  a  water  temperature  of  195  degrees  is 
dangerously  high  and  that  the  driver  must  cease  operations  and  stop 
the  engine.  This  is  incorrect.  In  fact,  temperatures  up  to  200 
degrees  are  considered  normal  for  these  trucks  under  heavy  work  con¬ 
ditions.  Additionally,  it  is  impossible  to  get  an  exact  reading  of 
195  degrees  from  the  gauge  while  200  degrees  can  be  read  with  pre¬ 
cision.  The  guidance  to  the  driver  should  be  to  check  his  vehicle 
and  report  to  maintenance  any  time  the  temperature  exceeds  the  normal 
range,  but  he  should  not  cease  operations  during  a  mission  until  the 
gauge  exceeds  200  degrees . 

The  lesson  teaches  that  when  the  vehicle  overheats  (195  degrees) , 
the  driver  should  cease  operations,  stop  the  engine,  and  wait  20  minutes 
After  waiting  20  minutes ,  he  should  check  the  water  -  if  the  water  is  OK 
check  the  oil  -  if  the  oil  is  OK,  report  the  condition  to  organizational 
maintenance.  The  20  minute  wait  is  necessary  to  allow  the  radiator  to 
cool  so  that  the  driver  is  not  burned  from  escaping  steam  and  so  that 
coolant  is  not  wasted.  There  is  no  need  to  wait  to  check  the  oil  level. 
As  a  matter  of  fact,  there  are  two  other  checks  the  driver  can  make 
immediately  that  will  be  more  probable  causes  of  overheating  (assuming 
the  before  operations  checks  were  made) .  These  are  checking  the  radi¬ 
ator  to  insure  the  radiator  fins  are  not  clogged  and  checking  to  insure 
that  the  fan  belt  is  not  loose  or  broken.  Therefore,  for  overheating, 
the  soldier  should  be  taught  to  stop  the  engine  and  immediately  check 
to  insure  the  fan  belt  is  OK,  the  radiator  is  clear  of  anything  that 
could  block  the  full  passage  of  air,  the  oil  level  is  correct,  and  then 
if  these  items  are  correct,  wait  20  minutes  and  check  the  coolant 
level . 

The  discussion  of  the  air  restriction  indicator  immediately  follows 
the  discussion  of  the  pressurized  air  system  that  operates  the  brakes. 

An  unknowledgeable  individual  may  assume  that  the  two  systems  are  the 
same.  The  lessons  should  make  it  clear  that  these  are  separate  systems. 

The  discussion  of  the  air  restriction  indicator  may  leave  an  unknow¬ 
ledgeable  individual  with  the  idea  that  the  air  restriction  indicator 


APPENDIX  C 


ADDITIONAL  OBSERVATIONS 


These  observations  and  impressions  are  not  directly  related 
to  the  validation  of  TEC  lessons,  but  are  considered  to  be  of  some 
significance  to  the  TEC  program. 

The  Printed  Text  Media. 

The  most  impressive  validation  observed  was  a  validation  of 
several  printed  text  lessons.  It  is  believed  that  the  media  selec¬ 
ted  contributed  significantly  to  this  impression  for  the  following 
reasons : 

o  The  lessons  addressed  message  formats  and  communica¬ 
tion  center  procedures  which  appeared  to  be  ideally 
suited  for  presentation  in  a  printed  text  format. 

All  personnel  in  this  MOS  are  required  to  read  and 
write. 

o  The  printed  text  format  allows  the  student  to  view 
the  lesson  in  the  final  format — a  true  one  one 
situation — and  provides  for  self-pacing  during 
validation. 

o  There  were  sufficient  personnel  (normally  five)  on 
hand  to  properly  control  the  validation  and  insure 
that  necessary  lesson  revisions  were  recorded. 

o  There  appeared  to  be  no  questions  from  a  doctrinal 
content  point  of  view.  The  lessons  addressed  well 
established  procedures  which  are  straightforward. 

The  lesson,  in  final  form,  is  readily  available  for 
wide  review. 

o  All  posttests  required  a  realistic  performance  of 
the  actual  task  being  taught  by  the  lesson. 

The  TEC  Managers  Guidebook  lists  the  relative  advantages  of  print¬ 
ed  text:  flexibility  in  lesson  design,  student  access,  mobility  (field 
use),  ease  in  updating/revisions,  and  low  cost.  The  relative  advantages 


of  audiovisual  are  listed  as:  student  interest/motivation,  inade¬ 
quate  reading  skills,  requires  visual  movement  presentation,  and 
presentation  of  psychomotor  and  perceptual  skills.  All  of  the 
eight  TEC  lessons  previously  observed  undergoing  validation  could 
have  been  presented  in  a  printed  text  mode.  Some  psychomotor 
skills  were  involved,  but  there  was  no  presentation  of  visual 
movement  in  any  lesson.  Most  of  the  information  could  have  been 
presented  well  within  the  reading  capability  of  the  target  audience. 
This  leaves  only  the  question  of  student  interest/motivation.  If 
the  interest  is  there,  will  the  soldier  be  more  motivated  to  visit 
the  learning  center  (perhaps  on  his  own  time)  to  view  the  audio¬ 
visual  lesson,  or  would  he  be  more  likely  to  pick  up  a  printed  text 
lesson  and  take  it  with  him  to  his  job  site?  The  printed  text 
also  has  some  operational  capabilities  which  the  Cue  See  does  not 
have — it  has  a  fast  forward,  a  fast  reverse,  a  repeat  and  looping 
capability,  and  the  heads  never  have  to  be  cleaned. 

The  Array  Training  System. 

If  the  primary  focus  of  training  should  be  on  combat  performance, 
there  is  little  evidence  that  adequate  consideration  has  been  given 
to  the  total  Army  training  system.  For  example,  the  training  system 
should  result  in  a  high  probability  that  soldiers  with  an  82C  MOS 
will  be  capable  of  performing  an  accurate  zone  to  zone  transformation 
when  the  task  is  required  in  combat.  A  detailed  analysis  of  this 
training  problem  would  probably  list,  as  first  priority  and  most 
cost  effective,  a  revision  of  DA  Forms  4212  and  4259.  An  audiovisual 
TEC  lesson  would  probably  be  on  the  bottom  of  the  priority  list. 

Selection  of  Subjects. 

An  overview  of  all  lesson  validations  and  a  general  sensing 
that  resulted  from  the  visits  indicate  that  there  has  been  a  push 
to  make  a  large  number  of  TEC  lessons.  It  appears  that  subjects 
have  been  listed  as  candidates  for  audiovisual  TEC  lessons  without 
adequate  concern  for  the  suitability  of  the  subject  matter  and  what 
the  usage  level  of  the  lesson  may  be. 

Use  of  Art  Work. 

Art  work  is  used  for  all  visuals  when,  for  many,  a  photograph  may 
better  accomplish  the  purpose  of  the  visual  -  and  be  much  cheaper. 
Additionally,  it  was  noted  that  at  least  two  schools  were  developing 
basic  operator  maintenance  lessons  for  the  M113  chassis.  It  appears 
that  much  of  the  art  work  would  be  applicable  to  all  lessons  that  in¬ 
volve  the  113  chassis. 


